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Abstract 

In the past two decades , numerous scheduling and load 
balancing techniques have been proposed for locally dis- 
tributed multiprocessor systems. Howe cr, they all suffer 
from significant deficiencies when extended to a Grid en- 
vironment: some use a centralized approach that renders 
the algorithm unscalable , while others assume the overhead 
involved in searching for appropriate resources to be neg- 
ligible. Furthermore , classical scheduling algorithms do 
not consider a Grid node to be N-resource rich and merely 
work towards maximizing the utilization of one of the re- 
sources. In this paper, we propose a ne w scheduling and 
load balancing algorithm for a generalized Grid model of 
N-resource nodes that not only takes into account the node 
and network heterogeneity , but also considers the overhead 
involved in coordinating among the nodes. Our algorithm 
is de-centralized , scalable, and overlaps the node coor- 
dination time with that of the actual processing of ready 
jobs, thus saving valuable clock cycles needed for making 
decisions. The proposed algorithm is studied by conduct- 
ing simulations using the Message Passing Interface (MPI) 
paradigm. 


1. Introduction 

Computational Grids [1,6] are typically a conglomera- 
tion of various resources with different owners, but make it 
possible for users to develop complex applications that ac- 
cess remote sites. Each of these sites (or nodes) could be a 
uni-processor machine, a symmetric multiprocessor cluster, 
a distributed memory multiprocessor system, or a massively 
parallel supercomputer. Each node consists of a number 
of heterogeneous resources ; the heterogeneity being in the 
type and capability of each of its A r -resources (e.g., number 
of processors, CPU speed, amount of memory, and so on). 
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Perhaps the biggest advantage of a heterogeneous Grid en- 
vironment over an isolated multiprocessor system is that it 
can offer resources to the user that are not locally available. 

With the Grid becoming a viable high performance com- 
puting alternative to the traditional supercomputing envi- 
ronment, various aspects of effective Grid resource utiliza- 
tion are gaining significance. With its multitude of re- 
sources, a proper scheduling and efficient load balancing 
across the Grid can lead to improved overall system per- 
formance and a lower turn-around time for individual jobs. 
Classical load balancing algorithms [3, 5, 14, 20] address 
this problem by maximizing the utilization of a single re- 
source (generally, CPU). But, the approach loses its merit 
for systems like the SUN Enterprise, the SGI Origin, and 
the IBM Regatta that offer multiple resources like shared 
memory, large disk farms, distinct I/O channels, and soft- 
ware licenses that can be independently allocated to differ- 
ent jobs. 

Another area where classical and even recent jV-resource 
load balancing approaches show their deficiency is in 
scalability — not many of them [10, 11, 12, 13, 14, 15, 18] 
can be scaled to the large number of processors in a Grid. 
This drawback is due either to the centralized approach of 
the algorithm [13, 18] or to the need for each node to have 
global system knowledge [11]. Also, most algorithms [10] 
either do not consider the overhead of searching for ap- 
propriate nodes or consider it to be negligible. This as- 
sumption is valid for tightly-coupled multiprocessor sys- 
tems [16, 17, 19], but not for geographically distributed en- 
vironments like the Grid. 

The present work is targeted to the Grid model where 
each node is assumed to be a A r -resource server and any job 
submitted to the Grid can be executed at any node. The only 
information our proposed algorithm needs before a node 
schedules a job is the communication latency between itself 
and its neighbors, thus making it fully scalable — an impor- 
tant consideration for a wide-area network like NASA’s In- 
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formation Power Grid (IPG) [2, 7]. The overhead involved 
in capturing the resource utilization status of a given node’s 
neighbors before making a scheduling decision can be a ma- 
jor issue negating the advantages of job migration. Our al- 
gorithm therefore overlaps the time spent looking for appro- 
priate nodes with the actual execution of the ready jobs, thus 
saving precious clock cycles. Also, since each Grid node 
(whether a single uni-processor machine or a multiproces- 
sor system) can have its own independent scheduling algo- 
rithm, our technique does not overrule the local schedulers’ 
job assignment policy. The class of problems we address 
is where jobs are computation-intensive and can be divided 
into totally independent sub-tasks with no communication 
between them. 

We have conducted extensive experiments using the 
Message Passing Interface (MPI) paradigm and by simulat- 
ing the job arrival rate. We compared the quality of load 
balance with the ideal case (where no overheads are in- 
volved) and found that our algorithm performs remarkably 
well in an heterogeneous Grid environment and gives en- 
couraging results. The remainder of this paper is organized 
as follows: Section 2 describes our algorithm and presents 
pseudo codes of the key procedures; Section 3 discusses the 
experimental setup that we used to test ar d substantiate our 
claims, and interprets the results; and Section 4 concludes 
the paper. 

2. Scheduling and load balancing 

Two important aspects of any wide area network sched- 
uler are its transfer [4, 15] and location [8, 9] policies. The 
transfer policy decides if there is a need to initiate load bal- 
ancing across the system, and is typically threshold based. 
Using workload information, it determines when a node be- 
comes eligible to act as a sender (transfer a job to another 
node) or as a receiver (retrieve a job from another node). 
The location policy selects a partner node for a job transfer 
transaction. In other w'ords, it locates complementary nodes 
to/from which a node can send/receive workload to improve 
overall system performance. 

Location policies can be broadly classified as 
sender- initiated [4, 21], receiver-initiated [4, 12], or 
symmetrically-initiated [5, 15, 19]. Sender-initiated 
policies are those w ? here heavily-loaded nodes search 
for lightly-loaded nodes while receiver -initiated policies 
are those where lightly-loaded nodes search for suitable 
senders. Symmetrically-initiated policies combine the 
advantages of these two by requiring both senders and 
receivers to look for appropriate partners. 

Load balancing policies can also be classified on the ba- 
sis of how up-to-date each node’s knowledge is about the 
state of the system. Dynamic [16, 17] policies make de- 
cisions based on the current system state and can rapidly 


adapt to workload fluctuations. On the other hand, policies 
that use static information and are not amenable to changes 
in the workload are knowm as static [3] policies. How- 
ever, dynamic policies incur the overhead of communicat- 
ing among the system nodes to keep them informed about 
the state of the system. 

In this section, we describe our scheduling and load bal- 
ancing algorithm for iV-resource Grid environments. It is 
dynamic, sender-initiated, and completely de-centralized. 
The last feature makes it extremely scalable for Grid en- 
vironments. A remarkable property of our algorithm is that 
it uses a smart search strategy for finding partner nodes. It 
also overlaps this decision making process of a node with 
the actual execution of ready jobs, thereby saving precious 
processor cycles. 

2.1. Preliminaries 

Before discussing the algorithm, let us introduce the con- 
cepts of Internal and External queues, w'hich we assume ex- 
ist in each Grid node. The Internal Queue of a node consists 
of the ready jobs which u'ould be executed by this particu- 
lar node only. Let r be the time when the tasks were last 
mapped, a(tj) be the arrival time of task tj, and e(tj) be 
the time tj starts executing. Then, the jobs in the Internal 
Queue are those that have been mapped and scheduled to 
this node, and are either being executed (Eq. 1) or are ready 
to be executed (Eq. 2); they w'ould never be delegated to any 
other node: 

{tj | a(tj) < t , e(tj) < r) (1) 

{tj | a(tj) < r, e(tj) > t } ( 2 ) 

Instead, the External Queue of a node consists of jobs which 
have been initially submitted to this node by a user, but are 
yet to be mapped and scheduled for execution (Eq. 3): 

{tj | a(tj) > r> e(tj) > t} (3) 

Let us now enumerate the key notations w'e will be using 
throughout the paper to explain our algorithm: 

• Pi \ Grid node i 

• Pi : The j-th resource of Pi 

• Jk : Job k 

• J j k : Ideal requirement for the j-th resource by J * 

• N eigh{Pi): Immediate neighbors of Pi 

• Compi(t): Time needed by P{ to empty its Internal 
Queue assuming no more jobs are assigned to it after time t 

• Comm j : Communication latency between Pi and Pj 

• ExQi'. Number of jobs in the External Queue of Pi 

We assume that each Grid node has knowledge about the 
communication latency between itself and all of its neigh- 
bors; i.e., each node Pi knows Comm V; € Neigh{Pi). 
Not only does this make the algorithm highly scalable, it 
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also allows the network to conveniently accommodate any 
changes in its topology. 

We also postulate that each incoming job knows its re- 
quirements for each of the resources available at a node. 
In order to generalize this concept, we define iV-resource 
jobs and A r -resource nodes/servers. Each job Jk looks for 
a node Pi with resources P^ 0 , P/, * . P, A ~ 1 , such that it 
meets its requirement for each resource type, J°, . .., 

J^ -1 . The algorithm described below r would be executing 
on every node of the Grid. 

2.2. Proposed algorithm 


Whenever a job is submitted by a user to a node Pj, pro- 
cedure Main (Fig. 1 ) invokes procedure NeedForTriggering 
(Fig. 2) to make a decision whether the job need to be mi- 
grated. If the job ought to be migrated to another node, 
a request is sent to all nodes j E Neigh(Pi), provided 
2 x Comm \ < TimejQ. This condition implies that the 
status request to the neighboring nodes and their responses | 
should be received before the Internal Queue is emptied | 
(denoted by TimejQ). This strategy avoids any wastage 
of the node's resources; the inequality overlaps the task of 
looking for appropriate nodes with the actual processing of 
the Internal Queue , thus hiding the overhead. 


Procedure Main 
Repeat forever 

If (a «— new job submitted) 

Time <— Current System Time (CST) 
NeedForTriggering (a, Time) 

If ( NeedForTriggering returns TRUE) 

TimejQ «— Time to empty Internal Queue 
V? G Neigh{Pi) 

If (2 x Comm\ < TimejQ ) 

Request { j 3 Comm \ , Time jq ) 

Receive (TimejQ ) 

Balance (S. R) 

End If 
End If 
End Repeat 
End Main 


Figure 1. Procedure Main 

Refer to Figures 2 and 3 for the triggering policy we have 
incorporated into our algorithm. It is based on the simple 
heuristic that greater the load at a node, the less inclined 
would it be to accept future loads. Within a time window 
of Compi(r ), triggering is initiated if the traffic burst is 
more than admissible; however, higher the resource usage, 
the smaller is the traffic burst that a node w ill accommodate 
(Fig. 3). 


Procedure NeedForTriggering ( a, T ime ) 

6 8 + a /* 8 is Cummulative Load */ 

If (CST -Time < Compi(r)) 

If {8 > Admissible Load at r) 

T <r~ CST 
Return TRUE 

Else 

Commit 8 to Internal Queue 
r CST 
8 <- 0 

Return FALSE 
End If 

End NeedForTriggering 


Figure 2. Procedure NeedForTriggering 



Figure 3. Value of 8 when Job Queue is x% full 


A node, having received a request to send the status of its 
resources, packs the information about their current utiliza- 
tion and sends it back to the requesting node along the route 
the request came (Fig. 4). This route is also piggybacked 
to the node w'hich needs to migrate load. Besides replying 
to requests, a node also recursively pings its neighbors for 
their resource status if its database says that the total round- 
trip latency between the sender and its neighbor would be 
less than TimeiQ. This allows the time required to look for 
additional resources be hidden under processing. 


Procedure Request (t, y 3 TimejQ ) 

Create Set S 

S. Route <— Route followed to reach i 
S.ResStatus <— Current usage of 

MPI_Send (S to i) 

Vj 6 Neigh(Pi) 

If { 2 x (7 4 * Commj ) < TimejQ ) 

Request (j, 7 + Comm { , TimejQ ) 

End If 

End Request 


Figure 4. Procedure Request 


3 






Figure 5 shows the pseudo code for procedure Receive. 
The sender waits for time TimeiQ to get replies from the 
nodes that have been queried for the status of their re- 
sources. 


Procedure Receive ( T imejo ) 
While { T ime < T ime / o ) 
MPIJReceive (S) 

End While 

R f- Number of rep', i as 

Return R 

End Receive 


Figure 5. Procedure Receive 


Figure 6 shows our procedure to schedule the jobs soon 
after TimejQ elapses. Without loss of generality, we can 
assume that 0.0 < P- , J j k < 1.0, 0 < j < N — 1. Let 
Mf be a match variable which defines the number of re- 
sources in node P, that fulfill the requirements of job J*. If 
bool(ji < P/) is 1 and bool(j£ > P/) is 0, then we can 
formally define M* as 


/v-i 

M k <- bool{J : k < Pf) (4) 

3 = 0 


Clearly, 0 < M k < N . Now, let us define matrices T 
and C, and vector V, as described in steps 1, 2, and 3 of 
procedure Balance (Fig. 6). Intuitively, the u-th row- and 
fc-th column of T gives the number of resources in node Pj 
that meets the requirements of job J*; the k - th entry of V 
gives the number of nodes which satisfy the minimum re- 
quirements of and element C u j denotes that there is a 
common node that satisfies the requirements of both J u and 
Jj , and that there might be a conflict while scheduling them. 

Another possible scenario is when the set of nodes that 
satisfy the requirements of J u is a subset of the set which 
satisfies the requirements of Jj \ in such cases, giving pref- 
erence to Jj might leave J u with no viable option. To 
avoid such cases, our algorithm first schedules jobs that 
have the fewest choices. P( Ujm m(Vj )) in step 4.1 of Fig. 6 
corresponds to the job Jmin that has the minimum num- 
ber of nodes it can be mapped to. The variable z indi- 
cates the node P z to which J m in can be delegated. Step 4.2 
checks matrix C and, in case there is another job that can be 
mapped to P z , chooses a different 2 for Jmin 3 if possible. 
Finally, Jmin is mapped and scheduled to P z . This mech- 
anism continues until all jobs have been scheduled or until 
no more can be mapped because of the lack of resources. 


3. Experimental study 

Here we describe the metrics used to gauge the perfor- 
mance of our scheduling and load balancing algorithm, the 
setup we had for our experiments, the simulation results that 
were obtained, and the conclusions we can draw from them. 

3.1. Performance metrics 


We analyze the performance of our algorithm using a 
parameter called Normalized Performance , rj (defined in 
Eq. 5). Basically, rj is the effectiveness of the load balanc- 
ing strategy. It is a comprehensive metric as it considers 
both the initial load balance as well as the load balancing 
overheads: 


T no - T my 

Tno “ Tib 


(5) 


Here, T no is the time to completely process all the jobs on a 
uniprocessor machine; To, is the time required by one pro- 
cessor divided by the total number of processors, thus pro- 
viding the runtime with ideal load balancing; and Tmy is 
the time needed by our algorithm to balance the load and 
execute all the jobs. Clearly, 


if T m y -» T lb: then 7 ] -4 1 (6) 

if T m y T no , then 77 -> 0 (7) 

These two conditions imply that higher the value of 77, the 
better is the load balancing; the ideal case being rj ~ 1. 


3.2. Experimental setup 


The experimental results reported in this paper were ob- 
tained by using an MPI implementation of our proposed al- 
gorithm. It is worth mentioning here that the various pa- 
rameters of our algorithm were varied following a Poisson 
distribution. Their respective mean values are given in Ta- 
ble 1. 


Table 1. Variables used in the experiments 


j Variables Mean 

Simulated by 

Processing Power 
Requirements 

2-16 

50 floatingpoint 
multiplications per unit 

Memory 

Requirements 

2-16 

1KB of memory 
allocated & freed per unit 

I/O Requirements 

2-16 

] KB of data written 
to disk per unit 

Network Latency 

5-11 

sleep(3) per unit 

Node Degree 

5 

number of 

neighboring nodes j 
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Procedure Balance [S, R) 

1. Using S, define matrix T of dimensions ExQi x R where T u ,k «— M* 

2. Define vector V of dimension ExQi where V* Ylu=i bool{T u> k = AT) , 1 < k < ExQi 

3 . Define matrix C of dimensions ExQi x where 

Ci,k Cfc.i f- 1 , if Tk,j = Tij = N ; 0 , otherwise ; 1 < Z, /c < ErQi , 1 < j < # 
4. Repeat until (no more ■ obs can be mapped) 

4.1 Z <- U | r( ttl mtn ( l'0) = :V ' l<u<R, 1 <j<ExQi 

4.2 If (QmtnCV^.fc) = 1 / l<fc<^*Qi) 

Choose another z t if possible 

4 . 3 Assign Jmin^Vj) to node P z , 1 < j < E’xQi 

4.4 Remove row mm(Tj' , 1 < j < and column 2 from T 

End Balance 


Figure 6. Procedure Balance 


Experiments were conducted for three different values of 
Max (15, 20, and 25) (see Fig. 3), and lepeated for 1-, 2-, 
and 3-resource nodes. The following three inequalities give 
the relationships between the mfs, where each m t * refers to 
the slope of the line joining the co-ordinates (0, Max ) and 
100, 0) (Fig. 3): 


m\ ,m 2, m3 < 0 

( 8 ) 

mi < m 2 < 777.3 

(9) 

mi 1 > 1 m2 | > | m 3 | 

( 10 ) 


3.3. Simulation results 

We have conducted extensive experiments to evaluate the 
performance of our algorithm and help us substantiate our 
approach. Figures 7 through 9 illustrate the results obtained 
from the study. 

To verify that our algorithm works well for completely 
heterogeneous systems, we divided the experiments into 
three groups. The first set of experiments was run on sys- 
tems where heterogeneity was in the capabilities of the 
A r -resources of a node; thus, the communication latency 
between all neighboring nodes was constant. The second 
set involved keeping the node capability constant and vary- 
ing only the communication latency between the nodes. Fi- 
nally, the third set of experiments combined the above two 
approaches, thereby exposing a totally heterogeneous setup 
to various load conditions (that were varied by changing the 
job arrival rate and the load associated with each job). Each 
set of experiments was repeated for 1-, 2-, and 3-resource 
nodes. The objective was to evaluate the algorithm thor- 
oughly by taking various scenarios of heterogeneity into 
consideration. 

Results for the first set (where only the capabilities of the 
A r -resources of a node are varied while keeping all other 
factors unchanged) are summarized by the graphs in Fig. 7. 
The horizontal axis represents the Mean Node Capacity of 


the network which can be defined as the mean value used 
for the capacity of each of the resources in a node (all re- 
source having the same mean). Increasing the resource ca- 
pability of the nodes without changing the job resource re- 
quirements effectively reduces the granularity of the latter. 
As depicted, any increase in node capability increases 77 . 
However, as the threshold slopes (mf s) become steeper, 77 
decreases. This is because the frequency of triggering the 
load balancing algorithm is reduced. 

In the second set of experiments, the Mean Node Capac- 
ity was held constant while varying the communication la- 
tency. The results presented in Fig. 8 show that 77 decreases 
with increasing communication cost. As in the previous set, 
the algorithm performs best when the absolute value of the 
threshold slope is the smallest (m3 in this case). 

For the final set of experiments, we vary the input load 
for a setup which has a heterogeneous mix of resource capa- 
bility and communication latency. This was repeated for 1-, 
2-, and 3-resource job specification for a 3-resource node. 
Figure 9 shows that the execution time decreases as we get 
more specific about job requirement. 

4. Conclusions 

In this paper, we presented a highly de-centralized, dis- 
tributed, and scalable algorithm for scheduling tasks and 
load balancing resources in heterogeneous Grid environ- 
ments. Our algorithm takes into consideration the over- 
heads of coordination and communication between the Grid 
nodes which were assumed to be N -resource servers that 
varied in their respective capacities across resources. The 
goal was to assign each node a job which would utilize 
its resources in the best possible manner, thus providing 
an effective scheduling and resource management strategy. 
We introduced a new load balance triggering policy based 
on the endurance of a node reflected by its current queue 
length. Also, our algorithm overlaps the time needed for 
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Figure 9. Execution Time ( T my ) vs. Average Load 


various communication overheads with that of executing the 
jobs already committed to the nodes, making the effective 
time for overheads virtually zero. The algorithm has been 
discussed in detail with pseudo codes being provided for all 
the major modules of the algorithm. 

To substantiate our claims, a comprehensive experimen- 
tal study was conducted using the Message Passing Inter- 
face (MPI) paradigm. Heterogeneity in resource capabil- 
ities and communication latency was maintained while re- 
peating the set of experiments for 1-, 2-, and 3-resource jobs 
and nodes. The Normalized Performance parameter was 
0.79 for 3-resource nodes and as high as 0.85 for 1 -resource 
nodes. These excellent performance levels could be attained 
only by overlapping the various overheads with the actual 
execution of the jobs. 
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